Feature/enhanced health #1676
Feature/enhanced health #1676
Conversation
- Implement real-time EFS mount health monitoring with I/O testing - Add multiple health endpoints (/healthz, /healthz/ready, /healthz/live, /healthz/mounts) - Integrate Prometheus-style metrics for monitoring systems - Prevent pod crash-loops through enhanced readiness/liveness checks - Add comprehensive test coverage and documentation This addresses the Kubernetes ecosystem need for modernized health probe logic to support proper CSI health monitoring, moving beyond basic HTTP responses to actual mount health assessment. Resolves: Enhanced health monitoring for production EFS CSI deployments
- Apply gofmt formatting fixes - Ensure consistent code style across all files - Ready for code review
k8s-ci-robot
added
the
cncf-cla: yes
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: oyiz-michael The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
k8s-ci-robot
added
the
needs-ok-to-test
|
Hi @oyiz-michael. Thanks for your PR. I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
k8s-ci-robot
added
the
size/XXL
|
/ok-to-test |
|
@oyiz-michael: Cannot trigger testing until a trusted user reviews the PR and leaves an In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
k8s-ci-robot
added
the
needs-rebase
|
PR needs rebase. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
#1675
Is this a bug fix or adding new feature?
Adding new feature
What is this PR about? / Why do we need it?
This PR implements enhanced EFS mount health monitoring to address critical production issues where pods crash-loop when EFS mounts degrade.
Problem: Current health probes only check if the CSI driver process is running, not whether EFS mounts actually work for I/O operations. This causes false positives where pods appear healthy but can't access storage.
Solution:
Real-time monitoring of EFS mount points with actual I/O testing
Multiple health endpoints (/healthz, /healthz/ready, /healthz/live, /healthz/mounts)
Prometheus-style metrics for observability
Enhanced CSI probe integration
Configurable health check intervals and timeouts
Benefits:
Prevents pod crash-loops by detecting mount issues before applications fail
Better observability with detailed mount health metrics
Kubernetes-native integration with readiness/liveness probes
Addresses issues #336, #1411, #1156.
What testing is done?
✅ Comprehensive unit tests for all health monitoring components
✅ Full driver test suite passes (31/31 tests)
✅ HTTP endpoint testing with proper status codes and JSON responses
✅ I/O testing validates actual file operations on mount points
✅ Code formatting (gofmt) and quality (go vet) verified
✅ Binary compilation successful
✅ Zero breaking changes - backward compatible